Add GizmoSQL #745
Add GizmoSQL #745
Conversation
|
|
|
Hi @prmoore77, thanks for your submission! However, I am having trouble getting it to run. When I run benchmark.sh I get the following output when creating the table (line 33): The only change I have made was adding 'sudo' to the docker run command since it wasn't working for me otherwise. Any ideas? |
|
Hi @george-larionov - could you share the |
|
Hi @george-larionov - I think I figured it out. In my AWS EC2 provisioning scripts (attached) - I mount the recommended EBS volume to path: Could you try with a volume mounted at that path? Here are the scripts: If you would like me to put the provisioning scripts in the codebase, just let me know. I didn't do so, b/c I didn't see that others had done so... Thanks for your help! |
|
hi @george-larionov - were you able to test? |
|
Hi @prmoore77, sorry for the late reply, it's been a busy week. I tried again after replacing /nfs_data with an existing path, but am still getting the same error, I don't think it is related to the mount path. The expectation is that the benchmark.sh script can run on a fresh Ubuntu machine on AWS, so please include any specific mounting you need (although perhaps it would be simpler to just use the default mount paths). Take a look at the CedarDB or DuckDB scripts. Additionally, the way the benchmarks are run is semi-automated using the run-benchmark.sh script, which fills in the cloud-init.sh script with the correct details and then runs it on the AWS instance, so take a look there for the exact way that things are run. I'm also attaching the log.txt file, is this the one you meant? Let me know if you have any questions, I will try to be more timely in my replies 😳 |
gizmosql/template.json
Outdated
| "column-oriented", | ||
| "arrow-flight-sql", | ||
| "duckdb", | ||
| "lukewarm-cold-run" |
There was a problem hiding this comment.
Please see here for more details what warm / lukewarm and cold runs mean in the context of ClickBench.
Two comments related to that.
run.sh contains this:
# Execute all queries in one session (so authentication overhead is minimized)
echo "Running benchmark with $(wc -l < queries.sql) queries, ${TRIES} tries each..."
gizmosqlline \
-u 'jdbc:arrow-flight-sql://localhost:31337?useEncryption=true&disableCertificateVerification=true' \
-n clickbench \
-p clickbench \
-f "${TEMP_SQL_FILE}"
The script does not clear the OS page cache between query runs - which it should do to qualify as a "lukewarm run". If authentication is costly, then feel free to disable it entirely (the script seems to disable certificate validation already).
The other thing is that we are moving away from lukewarm runs. New submissions ideally do cold runs right from the start. Would it be possible to kill/start the GizmoSQL docker container between query runs?
There was a problem hiding this comment.
hi @rschu1ze - would we need to kill/restart the container between each individual query execution?
There was a problem hiding this comment.
You would need to do this:
- kill/restart the container, then run query 1 three times (the first execution is a cold run, the second and third execution are hot runs),
- kill/restart the container, then run query 2 three times,
- kill/restart the container, then run query 3 three times,
- etc.
There was a problem hiding this comment.
Thanks for the clarification, I appreciate it. Sorry for my confusion - I'm a little new to this...
There was a problem hiding this comment.
I may switch to just using the gizmosql_server executable directly - to eliminate Docker from the test stack... It will make the installation just a tad more difficult - but not too bad...
There was a problem hiding this comment.
Thanks for the clarification, I appreciate it. Sorry for my confusion - I'm a little new to this...
It is no problem at all! We are always happy to expand the benchmark to new systems.
I may switch to just using the gizmosql_server executable directly - to eliminate Docker from the test stack... It will make the installation just a tad more difficult - but not too bad...
It is totally up to you, some DBs use docker (like CedarDB) while others install directly (like DuckDB).
|
hi @george-larionov and @rschu1ze , I have re-factored the benchmarking for GizmoSQL to use direct executables instead of Docker. I also stop/start the server between each query as directed, and I clear the system cache before running the benchmark. Could you re-review? Thanks! |
|
hi @george-larionov - have I addressed your concerns? Also - I wasn't clear - do you all run the script (via CI, etc.) for all of the platforms - so scores will be shown for all of them, or do I need to run it for the other machine types? |
|
Hi @prmoore77, I successfully reproduced the results locally (🎉 ), but still trying to get it to run with the automated script. I think I need to figure that out on our end but I will keep you updated. Regarding the scores for the other machines, yes, once we get your script running in our automation we will just run for multiple sizes, you don't need to. |
|
@prmoore77 here are the results from my run, just FYI (although, I will use the results from our automated run once I get it running for the final submission): |
|
@prmoore77 I have merged the branch to make it easier for me to debug why the automation isn't working. I will let you know once I update the actual benchmark results and add them to ClickBench. |
|
hi @george-larionov - are there any issues I can help with regarding the automation and getting the script to work? |
|
@prmoore77, not having much luck debugging the issue. It seems that when I run Archive: gizmosql.zip
inflating: gizmosql_server
inflating: gizmosql_client
/tmp /ClickBench/gizmosql
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
^M100 46.1M 100 46.1M 0 0 92.9M 0 --:--:-- --:--:-- --:--:-- 92.9M
/ClickBench/gizmosql
Waiting for gizmosql_server to start...(hangs here) And here is the gizmosql_server.log: It seems the underlying DuckDB needs some settings? Not sure... |
Hi @george-larionov - I think that the env var:
|
@prmoore77 that worked! |
4 participants
Closes #733
Thank You for Your Contribution!
We appreciate your effort and contribution to the project. To ensure that your Pull Request (PR) adheres to our guidelines, please ensure to review the rules mentioned in our contribution guidelines:
ClickHouse/ClickBench Contribution Rules
Thank you for your attention to these details and for helping us maintain the quality and integrity of the project.